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(54) Packet network transit delay measurement system 


(57) A system for analyzing and estimating average 
Transit delay and high percentiles of delay in a packet 
network (10) uses detailed packet switch measure* 
ments (30) that are normally stored in a network data- 
base for maintenance and engineering purposes. The 
system randomly selects source (12) and destination 
(14) points and samples the stored maintenance meas- 
urements (30) lor network circuit elements located be- 
tween the randomly selected source and destination 


points. These sampled measurements are transmitted 
" to the system processor via electronic mail and the sys- 
tem generates a single server queuing model (40) to de- 
termine average and variance of delay for each compo- 
nent. A network-of -queues model is then constructed to 
determine the end-to-end average delay and end-to- 
and variance of delay. An Erlang probability distribution 
is then applied to these end-to-end values to provide 
statistics of packet transit delay in the packet network 
(10). 
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Description 

BACKGROUND OF THE INVENTION 
s Bald of the Invention 

The present invention relates to data commimictions networks, and more particularly to a system for measuring 
transit delay in a packet network. 

io Description of the Related Art 

In many data communications networks, data is generated in high intensity bursts that are often separated by idle 
periods therebetween. The average data rate may, therefore, be much lower than the peak rate. An economic way of 
transmitting this type of intermittent (bursty) data is to assemble the data into what id known in the art as "packets' and 
is interspersing packets from several different channels on a singls. physical communication path. This communication 
path is therefore occupied by a particular packet only for the time corresponding to the packet length, and the com- 
munication path is then available for use by other packets that may be betweon different data terminals. 

A packet refers to a group o* bits that are switched in the network as an integral unit, A packet may also include 
a header that is used to identify that packet. A packet is formatted in a particular way and may contain data, and 
20 destination, origination and control information. 

In a typical packet-switching network, a number of users can be interconnected. Packet switches are utilized In 
the network to sort packets received from one circuit and direct these packets to another circuit according to the header 
information for each packet A typical packet switch is AT&T's 1 PSS™ packet switch which implements the CCITT X 
25 packet-switching protocol. 

as in any data communications network, a primary concern in me performance of the network is the time in which 
information is processed and transmitted back to the user. This timis is typically referred to as response time or network 
transit delay. Network transit delay in a packet-switching network may be a function of data transmission speed, prop- 
agation time, distance through the network, packet-switch processing time and queuing delay. 

To evaluate the performance of a packet network, transit delay is generally measured from the time the last bit of 

so a particular packet enters the provider's originating central office equipment (e.g., originating packet switch) to the time 
the first bit of the packet leaves the provider's destination central office equipment (e.g., destination packet switch). 

In designing and maintaining data communications systems, it is essential to minimize network transit delays. In 
fact, most customers today have strict requirements regarding the permissible transit delay in me network. Customers 
will often require the provider of the data communicattons network to pay a penalty if the transit delay exceeds a 

ss predetermined level. 

Therefore, H is important for the design and maintenance of a data communications system to effectively measure 
transit delay in the packet network. Moreover, customers ol the system frequently require periodic measurements of 
transit delay to ascertain whether the system is performing within the limits of the customers' service contracts. 
There are currently a number ol measurement systems used today to estimate or measure network transit delay 
40 These measurement systems generally fail Into two categories and are illustrated in FIG. 1 . 

The first type of measurement system is a protocol analyzer 20 which includes hardware connected to the packet 
network 1 0 for analyzing and measuring transit delay Transit delay is measured by connecting a protocol analyzer 20 
to a desired measurement point in the network 1 0 and essentially time stamping a particular packet to determine the 
actual transit delay of that particular packet 
45 This type of measurement system, however, can be extremely expensive because it requires a protocol analyzer 

20 at each desired measurement point. Thus, if a customer has numerous access points to the network 10, as they 
Often do, protocol analyzers 20 would be required at each point Of access into the network 1 0. 

in addition to the expense associated with requiring a protocol analyzer 20 at each desired measurement point 
each protocol analyzer 20 must be located in the service node ft*, the packet switch) in the network 10. and also 
so requires space for locating the protocol analyzer 20, power for operating the equipment, and the ability to access the 
equipment tor installation and maintenance purposes. 

The second type of system tor measuring transit delay is through the use of measurement software 22 that may 
exist in the customers end equipment and/or at various locations in the packet network 1 0. The measurement software 
22. however, must be developed, tested and integrated into the network 10 and frequently requires updating (remte- 
55 gration) each time the software in the packet network 10 is updated, changed or otherwise modified (software main- 
tenance). 

It is, therefore, desirable to provide a cost effective, noninvasive measurement system to estimate transit delay 
in a packet network 10 that ctoes not require hardware or measuring software to be added to the network 10. 
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fti IMMARY OF INVENTION 

The present invention provides a cost effective system for estimating tha end-tcnend packet delay distributions in 
a packet network. The system analyzes and estimates average transit delay and high percentiles of delay in a packet 

s network by utilize detailed packet switch measurements that are normally collected and stored in a network database 
for maintenance and engineering purposes. The system randomly selects various source and destination points and 
samples the stored maintenance measurements tor specific network circuit elements between the randomly selected 
source and destination points. These sampled measurements are transmitted to the system processor and a single 
server queuing model is Generated to determine average and variance of delay tor each identified element A network- 

iQ ol<iueues model is then constructed to determine the end-to^end average delay and end-to-end variance of delay. An 
Erlang probability distribution is then applied to these end-to-end values to provide statistics Of packet transit delay in 
the packet network. 


35 


40 


BRIEF DESCRIPTION OF THE DRAWINGS 


PIG. 1 is a network overview illustrating the prior art methods of measuring packet network transit delay. 
FIG. 2 is an illustration of the measurement modelling approach of the present invention. 
FIG 3 is an illustration of a preferred embodiment of the present invention. 

FIG. 4 illustrates the networks-queues model utilized in the preferred embodiment of the present invention. 
20 FIGS. 5-7 are block diagrams of steps performed in the preferred embodiment of the present invention. 

DESCRIPTION OF THE PREFERRED EMBODIMENTS 


The present invention provides a cost effective method for estimating the end-to-end delay distributions n a packet 
£S network Referring to FIG. 2. a packet network 1 0 is generally shown connecting a data source 1 2 to a data destination 
14 Although not illustrated, it is commonly known that a packet network 10 may consist of numerous elements or 
components, including, but not limited to, access lines, gateway links, inter-switch trunks, packet switches, packet 
assembler and disassemblers (PAD), concentrators, multiplexers and interlaces. 

AS part Of the normal operation of a packet network 10; it is common for a proper of the network 10 to collect 
W certain packet switch maintenance measurements or data 30 tor engineering and maintenance purposes to keep the 
network 10 operational. Trie maintenance measurements 30 are typically stored in a database 32 of network 10 as 
illustrated in FIGS. 2 and 3. The maintenance measurements 30 also represent the amount of packets that are trans- 
mitted through the network 10, ^ ¥ 

For each circuit in the network 10, these maintenance measurements 30 commonly include packet counts, oyie 
counts speed of circuit, errored Trame rate, average packet size, packet retransmission rate, packet switch Ration 
and circuit utilization. Although wall known in the art, the above^nentioned maintenance measurements 30 will be 
generally described below. 

A packet count generally refers to the number of packets switched by a particular element (i.e., a packet switch) 
in the network 10 m a given period of time. Byte count refers to the number of bytes contained in the .nformation field 
of the packets that have been switched by the particular element in the network 10 in a specified interval of time. 

Speed of Circuit is a fixed parameter that generally refers to the capacity of the network circuit ,n bits per second. 
Errored frame rate refers to the number of frames (data link layer) that were received in error in a <*ven nerval of time. 

Average packet size generally refers to the average size of packets measured in bytes, Packet retransm.ssran 
rate refers to the number of packets retransmitted (e.g.. transmitted more than once) in the network 1 0 m a given period 

°* ^Packet switch utilization refers to the fraction ol the capacity of the packet switch processor (not shown) that is 
utilized in a given interval of time. Packet switch utilization is based upon ihe previously described packet count. 

Circuit utilization refers to the fraction of thecapaeiryof me netv^rkci^ 
period of time Circuit utilization is derived from the previously described byte count. 

As is illustrated in FIGS. 2 and 3. the present invention utilizes the above-mentioned maintenance measurements 
30 for specflic network elements to generate a mo^^ 

the end-to-end delay in the network 10. , _ _ 

The sequence Of operation of the preferred embodiment of the present invention is illustrated in HGS 5-7. Trie 
maintenance measurements 30 in network 10 are collected (step 100) and stored (step n0) in database 32. As pre- 
viously stated, steps 100 and 110 are commonly performed in packet networks 10. . . _ „ . B 

Since a typical packet network 10 may be connected to multiple source points 12 and dest-nauon points 14 t is 
necessary to identify a particular source point 12, destination point 14, date and Hme of measurement in order to 
determine the packet network transit delay (step 120). This ratification may be accomplished by user input 
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In the preferred embodiment ot the present invention, however, it is preferable to randomly sample different points 
in the network 10. The preferred embodiment, therefore, randomly selects a desired source point 12 and destination 
point 14 for measuring the network transit delay therebetween, and randomly chooses ihe date and time of such 
measurement. , 
s By randomly selecting the source and destination points 1 2, 1 4, and date and time of sampling, the present inven- 

tion t© capable of generating a model 40 of the network 10 and need not take actual measurements between every 
source and destination point 12,14 connected to the network 1 0. This is desirable because, for every N end points in 
network 10, there exist end-to-end points. Thus, if there were 5000 end points in network 10, there would be 25 
million paths to measure. Assuming that a customer desired hourly network transit delay measurements (also referred 
10 to as quality of service (QOS) measurements), this would amount to approximately 20 billion QOS measurements per 
month. Selecting random end-to-end paths based upon randomly selected source and destination points 12. 14 and 
taking measurements at random times dramatically decreases the number of QOS measurements necessary to gen- 
erate me model 40 of me network 1 0. 

Although the preferred embodiment of the present invention samples the packet network transit delay between 
;$ randomly selected source and destination points 1 2. 1 4, it is understood that this method is utilized to make the system 
more practical. However, it is not a prerequisite of the present invention to randomly sample network transit delay 
While it may be unpractical or inefficient, the present invention may also be utilized to actually measure transit delay 
based upon each source and destination point 12, 14 connected to network 10. 

Once the desired source and destination points 1 2, 1 4 are identified in step 1 20, the system identifies the packet 
so . switches in network 10 that each end point is connected to (step 1 30). The network packet switches are determined 
based upon the particular source and destination points 1 2, 1 4 previously identified in step 1 20. 

> In step 140, the system also identifies the trunks and tandem switches (if any) between the previously ictentfied 
source and destination points 12, 14. Having determined the desired source and destination points 1 2. 1 4 in step 130 
the present invention utilizes known routing tables in step 140 to identify the trunks and tandem switches in the network 
& 10 between the source and destination points 1 2, 1 4. Thus, tor each desired source and destination point 1 2, 14, the 
system identifies the trunks and associated packet switches in network 10 connecting the desired source and desti- 
nation points 12, 14. . t . 

Having identified the associated packet switcnes in network 10 that connect the desired source point 1 2 to the 
desired destination point 14, the system then reads (step 150) the relevant maintenance measurements 30 collected 
so and stor ed in the network database 32 (steps 1 00, 110) for each identified packet switch corresponding to the identified 
source and destination points 1 2, 14. transports (step 1 60) the relevant maintenance measurements 30 to the modelling 
processor 50. and feeds (step 170) the relevant maintenance measurements 30 into the network modelling program 
operating in the modelling processor 50 as illustrated in FIG. 3. Methods well known in the art to transfer such data 
Include, but are not limited to. electronic mail systems or file transfer protocol 
35 By utilizing the relevant maintenance measurements 30, the modelling processor 50 estimates end-to-end delay 

distributions for packet network 10. The system generates a model 40 of the network 10 by generatmg single server 
M/G/1 queuing models for individual network elements (e.g., identified packet switches) and constructing a network- 
ot-queues model in a series as illustrated in FIG. 4. For analytical treatability, all queues are assumed to be independent. 
As will be discussed below, an Erlang probability distribution is then fit to two end-to^nd delay parameters (average 
40 and variance of delay) to predict service delay performance of the network 1 0. 

Thus, for each network element (N), the system generates a network element single server queung model. Each 
element model has two primary output parameters: average transit delay (mj and variance of packet delay The 
average transrt delay (rrt) is broken down into two components, namely, a constant (non-variable) component (mj ) 
and a variable component (m ; v). The variable component (mf) is derived from the randomly variable portion of the 
4£ packet delay 

Referring to step 180 in FIG. 6, me average transit delay (rnj) for each network element (N) is calculated using the 
following well known formula for an M/G/1 queuing model: 

(1 + cj) 
m i= x /( 1+ P- , 5(T^P? ) 

where x, represents average service time, p { server utilization, variance of service time, and Cd*9jt&& (the 

coefficient of variation). ' . ... 

Referring now to step 190 in FIG. S, the variance ot packet delay is determined using the following well known 

lormute reran M/G/1 queuing model; 

$5 


SO 
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25 


30 


40 


Gi 1 2(1- Pi ) ] 


pi 


3- "Pi 3*, 


where xj 5 represents the third moment of service time. i- t average service time, p t server utilization, variance of 
service time, and C^a^/fiif (the coefficient of variation). 

Once the average transit delay (mj) and variance of packet delay (aj 2 ) are determined for each network element 
(N) in steps 180 and 190, end-to-end performance measures for network 10 are obtained in step 200 by addition of 
each m, and a*, resulting in end-to-end average delays (both fixed (m«) and variable (m v )) and end-to-end variance 
of dalay (o 2 ). 

The detail* of step 200 are given below: Let rrij and c? denote the average and variance of delay of the ith network 
element (N). Let mf and mf denote the constant and variable portions of the average delay, respectively. Accordingly, 
the end-to-end average (m) and variance (cr 2 ) packet delay for the network 10 is determined as follows: 


m 


J>i. 


N 


Once step 200 is completed and the end-to-end ave rage packet delay (m) and variance of delay (o*) are detemn ined 
for network 10, a simple Erlang probability distribution is applied to the end-to-end average and variance of delay m 
step 21 0. thereby providing the high percentiles of delay (e.g.. 95th percentile) frequently required by customers. Only 
the variable portion of the average delay (m v ) and the variance of delay (o 2 ) are used in the Erlang dstribution. The 
fixed portion of the average delay m* is excluded to avoid biasing the fit through effects of propagation delay and 
constant components of processing or insertion delays. The bias tends to make the Erlang distribution have high order, 
which may cause under-estimation of the high percentile of delay. 

The Erlang model requires two parameters, namely, average and order (r). The average is simply set to m v . while 
the order <r)=min[rnax[l ,L (mv/<y)2j],i 0]- The order (rj is limited to values between 1 and 1 0. The value of 1 corresponds 
to an M/M/i queue, while the maximum value of 10 ensures that a certain minimum variability is maintained in the 
model 40. 

The resulting Erlang distribution as a function of delay (d) is denoted Efrn* r,d). The pth percenfcle of delay corre- 
sponding to a given probability (p) is denoted E" 1 (mV r,p). The network delay performance measures are then simply 
estimated as follows: 


6Q 


55 


Average Delay: m 

pth Percentile of Detey: E*HrTrlV.p)+m* 

The table below shows the relationship among E'MmV.p), r and p for various example values of r and p. Note that 
for given values of r and p, E -1 (mV.p) is proportional to m\ 


r 

p^0-95 

p=Q.90 


95th •fctiteMl* 

90th %tile/m v 

1 

3.0 

2.4 

2 

2.4 

1.95 

3 

2.1 

1.8 

4 

i.gs 

1.7 
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(continued) 


5 


r 

p=0.95 

p=0.90 


95th SMile/m* 

90th %tile/m v 

5 

1.85 

1.SO 

6 

1.S 

1.55 

7 

17 

1.55 

a 

1.65 

1.5 

9 

1.65 

1.45 

10 

' 1.60 

1.45 


One© the high percentiles of delay are determined in step 210. the system outputs the results as illustrated in Step 
220 of FIG. 7. In the event that all data has not been read (step 230), the system repeats steps 150 through 220 until 
all ot the desired data has been processed. When all of the data has been read in step 230. the system generates a 
complete QOS summary report (step 240) detailing the performance of the network 10 with respect to its packet transit 
delay objective. 

It is to be understood that the above description is only of a preferred embodiment of the present invention. Nu- 
merous other arrangements may be devised by one skilled in the art without departing from the spirit and scope of the 
invention. The invention is thus limited only as defined in the accompanying claims. 


2$ 


Claims 


33 


t . A packet network transit delay measurrig system for estimating packet transit delay in an existing packet network 
having a plurality of packet swBches. comprising the steps of: 

(a) collecting packet switch measurements; 

(b) determining element packet delay statistics based upon said collected packet switch measurements; 

(c) determining end-to-end packet delay statistics from said element packet delay statistics; 

(d) randomly selecting a source point and a destination point to sample packet transit delay therebetween; 

(e) identifying the packet switches located on a circuit between the randomly selected source and destination 
points; and 

wherein the element packet delay statistics are determined for each identified packet switch based upon col- 
lected measurements for that identified packet switch. 

2. A system as claimed in claim 1 , further comprising the step of storing the collected packet switch measurements 
40 in a network database. 

3. A system as claimed in claim 2. wherein said collected packet switch measurements are transmitted from said 
network database to a processor means for determining element and end-lo-end packet delay statistics. 

43 4. A system as claimed in claim 1 . wherein the packet switches located on the circuit between the randomly selected 
source and destination points are identified using routing tables. 

5. A system as claimed in claim 3, wherein said collected packet switch measurements are transmitted from said 
network database to said processor means by electronic mail. 

so 

6. A system as claimed in claim 3, wherein said collected packet switch measurements are transmitted from said 
network database to said processor means by file transfer protocol. 

7. A system as claimed in claim 1 . wherein the determination of said element delay statistics for each identified packet 
68 switch comprise the steps of; 

(a) determining average packet transit delay based upon said packet switch measurements; and 
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(b) determining variance of packet delay based upon said packet switch measurements. 

8. A system as claimed in claim 7, wherein the determination of said end-to-end delay statistics comprise the step 
of summing the average packet transit delay and variance oJ packet delay tor each identified packet switch. 

5 

9. A system as claimed in claim 8. wherein high percentiles of delay statistics are determined by applying an EHang 
probability distribution to the end-to-end delay statistics. 

10. A system for estimating packet transit delay in an existing packet network having a plurality of packet switches 
w said system comprising: 

(a) means for collecting and storing maintenance measurements tor said packet switches; 

(b) processing means for determining element and end-to-end packet delay statistics based upon said [packet 
switch] collected and stored maintenance measurements; and 

is (c) means for transmitting said stored maintenance measurements from said collecting and storing means to 

said processor; and wherein said processing means; 

(d) randomly selects a source point and a destination point to sample packet transit delay therebetween; and 

(e) identifies the packet switches located on a circuit between the randomly selected source and destination 
points. 

20 

11. A sysiem as claimed in claim 10, wherein the element packet delay statistics are determined tor each identified 
packet switch based upon collected measurements for that particular identified packet switch. 

1 2. A system as claimed in claim 10, wherein the packet switches are identified using routing tables. 

25 

13. A system as claimed in claim 12, wherein said element delay statistics comprise: 

(a) average packet transit delay for each identified packet switch determined from the transmitted measure- 
ments tor each particular identified packet switch; and 
30 (b) variance of packet delay for each identified packet switch determined from the transmitted measurements 

for each particular identified packet switch. 

14. A system as claimed in claim 13, wherein said processor determines said end-to-end delay statistics by summing 
the average packet transit delay and variance of packet delay tor each Bentified packet switch. 

35 , 

1 5. A system as claimed tn claim 1 4, wherein said processor determines high percentiles of delay statistics by applying 
an Erfang probability distribution to the end-to-end delay statistics. 
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FIG. 6 
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Defective images within this document are accurate representations of the original 
documents submitted by the applicant. 

Defects in the images include but are not limited to the items checked: 

□ BLACK BORDERS 

□ IMAGE CUT OFF AT TOP, BOTTOM OR SIDES 

□ FADED TEXT OR DRAWING 

□ BLURRED OR ILLEGIBLE TEXT OR DRAWING 

□ SKEWED/SLANTED IMAGES 

□ COLOR OR BLACK AND WHITE PHOTOGRAPHS 

□ GRAY SCALE DOCUMENTS 

□ LINES OR MARKS ON ORIGINAL DOCUMENT 

□ REFERENCE(S) OR EXHIBIT(S) SUBMITTED ARE POOR QUALITY 

□ OTHER: 

IMAGES ARE BEST AVAILABLE COPY. 
As rescanning these documents will not correct the image 
problems checked, please do not report these problems to 
the IFW Image Problem Mailbox. 


